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Outline 


• Highlights from April 7 Net-Meeting presentation 

- “Comparison of results run at JPL using different Start-up options" 

• Further results related to Start-up options 

• Comparison of JPL 2 Regression MODIS with SRT Version-5.44 

- SRT Version-5.44 is functionally equivalent to JPL 2 Regression MODIS 

with minor differences 

• Improved cloud parameter retrievals using SRT Version-5.44 

• Future plans for Version-6 at SRT 
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Highlights from Net-Meeting 
Experiments We Have Run at JPL 

All experiments used JPL Version-5.7.4 with three different start-up options 
Version-5.7.4 Baseline MODIS (two regression) 

Version-5.7.4 SCCNN 
Version-5.7.4 Climatology Physical 


All experiments used MODIS 10 point emissivity initial guess over land 
Each experiment was run in the AIRS/AMSU mode and in the AIRS Only mode 
Each experiment was run for the same 6 days we use for experiments run at SRT 
September 6, 2002 
January 25, 2003 
September 29, 2004 
August 5, 2005 
February 24, 2007 
August 10, 2007 

May 30, 2010 added per request of Evan Manning 


Validation is performed using colocated ECMWF as “truth” on 6 days 
Trends include seven days as requested by Evan Manning 
We have generated separate error estimate coefficients and QC thresholds to be 
used for, and only for, each experiment 
We present results of QC’d T(p) and SST 
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Methodology Used for T(p) Quality Control in Version-5 


Define a profile dependent pressure, p best , above which the temperature profile is 
flagged as best - otherwise flagged as bad 

Use error estimate 5T(p) to determine p best 

Start from 70 mb and set p best to be the pressure at the first level below which 
5T(p) > threshold AT(p) for 3 consecutive layers 

Temperature profile statistics include yield and errors of T(p) down to p = p best 

Version-5 used AT(p) thresholds optimized simultaneously for weather and 
climate : A"P tandard (p) 

Subsequent experience showed AT standard (p) was not optimal for data assimilation 
(too loose) or for climate (too tight) 

Use of new tighter thresholds AT t '9 ht (p) resulted in retrievals with lower yield but with 
RMS errors »1K 

Tight QC performed much better when used in data assimilation experiments 

Standard QC performed poorly in the lower troposphere over land 
Standard QC defined cases with QC=0 in Version-5 
A kluge was needed over land to generate cases with QC=1 
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Methodology Used for T(p) Quality Control in Version-6 


Essentially no retrievals are “left behind” 

QC is applied to all cases in which a successful retrieval is performed 

All successful retrievals have QC=0 down to 30 mb 

QC is otherwise analogous to Version-5 but has tight thresholds AT A (p) for data 
assimilation and loose thresholds AT c (p) for climate applications 

AT a QC thresholds define p best (QC=0) and AT C thresholds define p good (QC=0,1 ) 

AT a QC thresholds were set for each experiment so as to give RMS 
errors «1 K 

AT C QC thresholds are used to generate level-3 gridded products 

AT C QC thresholds were set for each experiment so as to maximize coverage and 
achieve < 2K tropospheric RMS errors 
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Performance Metrics 


We evaluate each start-up option in terms of accuracy as a function of % yield 
We compare yields and RMS errors for each experiment using their own QC 
thresholds 

Ability to do effective QC is critical for a given system 

We also compare RMS errors for each experiment using 2 common sets of cases 

1) All cases accepted by Version-5 Tight QC 

How do start-up options compare on less challenging cases? 

2) All cases accepted by SCCNN climate QC 

How much do start-up options degrade under challenging but doable 
cases 

Tropospheric Temperature Metric (TTM) is the average RMS error for all 1 km 
layers between 1000 mb and 100 mb 

Yield Metric (YM) is the average % yield for all 1 km layers between 1000 mb and 
100 mb 

A start-up option must perform well in the AIRS Only mode to be acceptable for 
Version-6 

A start-up option must also result in minimal yield and temperature bias trends 
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Comparisons Shown 


We first compare Version-6 SCCNN and SCCNNAO with Version-5 Tight and 
Version-5 Standard 

We then compare Version-6 Regression, Climatology, and SCCNN with each other, 
including AO runs 
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Differences from ECMWF 
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Global 
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V5 Standard Thresholds (0C=0) 

Neural Net OA OC (QC=0, PBest) 

Neural Net Climate OC (QC=0,1; PGood) 

Neural Net AIRS Only OA OC (QC=0, PBest) 

Neural Net AIRS Only Climate OC (QC=0,1; PGood) 
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N u m b e r 


Surface Skin Temperature Difference 
6-Day Daytime and Nighttime combined 
50 N to 50 S Non-Frozen Ocean 
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Comparison of Version-6 Neural-Net with Version-5 


Version-6 Neural-Net performs significantly better than Version-5 in all regards 

Temperature Profile 

• Yield using Data Assimilation QC is much greater than Version-5 tight with 

comparable RMS errors 

• Yield using Climate QC is much greater than Version-5 standard with good 

RMS errors 

• Lower tropospheric Neural-Net retrievals have comparable or better accuracy 

than Version-5 for less challenging cases 

• Version-5 retrievals degrade much faster than Neural-Net retrievals for difficult 

cases 

• Improvement over Version-5 is largest over land 
Bias Trends 

Neural-Net yield and spurious bias trends are significantly better than Version-5 
Sea Surface Temperature (SST) 

Neural-Net SST’s have significantly higher yields and better accuracy than 
Version-5 


Neural-Net AO retrieval performance is only marginally poorer than Neural-Net 
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Tropospheric Temperature Performance Metric Using Own Data Assimilation Thresholds 



Global 

YM(%) TTM(K) 

Land ±50° 
YM(%) TTM(K) 

Ocean ±50° 
YM(%) TTM(K) 

Poleward of 50°N 
YM(%) TTM(K) 

Poleward of 50°S 
YM(%) TTM(K) 

Version-5 Tight 

46.2 

1.08 

42.0 

1.17 

60.9 

1.02 

35.9 

1.15 

31.2 

1.30 

Neural-Net 

70.9 
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1.20 

2 Regression 
MODIS 
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Tropospheric Temperature Performance Metrics Using Own Climate Thresholds 



Global 

YM(%) TTM(K) 

Land ±50° 
YM(%) TTM(K) 

Ocean ±50° 
YM(%) TTM(K) 

Poleward of 50°N 
YM(%) TTM(K) 

Poleward of 50°S 
YM(%) TTM(K) 

Version-5 Standard 

70.3 

1.25 

70.2 

1.34 

72.6 

1.07 

69.3 

1.30 

66.0 

1.45 

Neural-Net 

93.4 

1.12 

91.5 

1.06 

96.7 

1.04 

90.8 

1.16 

90.9 

1.31 

2 Regression 
MODIS 

83.8 
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1.15 
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1.42 

78.6 
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Climatology 

79.4 
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83.9 
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1.40 

79.5 

1.22 

69.6 

1.43 

54.6 

1.48 

Climatology AO 
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1.33 
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1.40 

78.2 

1.25 

67.3 

1.42 

54.7 

1.41 
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Further Results Related to Start-up Options 


1) Results shown at April Net-meeting for 6 days using ensembles in common were 
incorrect. They did not contain all 6 days. We have corrected plots and tables. 

2) New table showing Boundary Layer Metric for common ensembles. 

Boundary Layer Metric is the average RMS difference from ECMWF for the four 
lowest of the 100 layers above the surface (1 km). 

N.B. These are 0.25 km layers. 

3) Results shown for cases in common include Neural-Net guess and Version-5 

Clear Regression guess 
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TTM (BLM) Metric Using the Version-5 Tight Ensemble 



Global 

Land ±50° 

Ocean ±50° 

Poleward of 50°N 

Poleward of 50°S 

Version-5 

1.08 (1.27) 

1.17 (1.69) 

1.02 (1.11) 

1.15 (1.49) 

1.30 (1.74) 

Neural-Net 

0.93 (1.18) 

0.95 (1.53) 

0.87 (1.00) 

1.00 (1.51) 

1.19 (1.73) 

2 Regression 
MODIS 

1.09 (1.34) 

1.12 (1.80) 

0.99 (1.16) 

1.20 (1.60) 

1.36 (1.81) 

Climatology 

1.18 (1.73) 

1.17 (1.94) 

1.11 (1.53) 

1.35 (2.11) 

1.47 (2.51) 

Neural-Net AO 

0.96 (1.34) 

0.99 (1.70) 

0.88 (1.14) 

1.05 (1.76) 

1.27 (1.91) 

2 Regression 
MODIS AO 

1.12 (1.37) 

1.16 (1.87) 

1.02 (1.20) 

1.22 (1.60) 

1.42 (1.81) 

Climatology AO 

1.10 (1.36) 

1.16 (1.80) 

1.03 (1.21) 

1.19 (1.57) 

1.32 (1.79) 
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TTM (BLM) Metric Using the Neural-Net Climate Ensemble 



Global 

Land ±50° 

Ocean ±50° 

Poleward of 50°N 

Poleward of 50°S 

Version-5 

1.62 (2.28) 

1.72 (2.43) 

1.58 (2.16) 

1.50 (2.15) 

1.73 (2.55) 

Neural-Net 

1.13 (1.75) 

1.07 (1.84) 

1.05 (1.38) 

1.17 (2.02) 

1.33 (2.22) 

2 Regression 
MODIS 

1.61 (2.84) 

1.50 (2.58) 

1.54 (2.62) 

1.62 (3.09) 

1.84 (3.33) 

Climatology 

1.44 (2.38) 

1.36 (2.35) 

1.30 (1.88) 

1.58 (2.70) 

1.66 (3.16) 

Neural-Net AO 

1.24 (2.07) 

1.15 (2.02) 

1.10 (1.58) 

1.34 (2.67) 

1.49 (2.57) 

2 Regression 
MODIS AO 

2.41 (4.59) 

2.30 (3.69) 

2.68 (5.27) 

1.98 (3.90) 

2.15 (3.82) 

Climatology AO 

2.60 (4.57) 

2.51 (3.96) 

2.98 (5.20) 

2.07 (3.84) 

2.12 (3.78) 
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Comparison of Version-6 Neural-Net Start-up with 2 Regression and 

Climatology 


Version-6 Neural-Net performs significantly better than other start-ups 

Temperature Profile 

• Neural-Net Yield using Data Assimilation QC is much greater than either other 

start-up with better RMS errors 

• Neural-Net Yield using Climate QC is much greater than either other start-up 

with significantly better RMS errors 

• Neural-Net retrievals degrade more slowly than other start-up retrievals for 

difficult cases in common 

• Climatology start-up performs poorer than 2 Regression for less challenging 

cases in common 

• Climatology start-up performs better than 2 Regression for difficult cases in 

common - climatology start-up degrades more slowly 

• Neural-Net AO retrieval performance is only marginally poorer than Neural-Net 

using AIRS/AMSU 

• 2 Regression and Climatology systems degrade significantly in AO mode for 

harder cases 
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Comparison of Boundary Layer Temperatures 


Comparisons done on common ensembles 

Easier cases selected using Version-5 Tight QC 
Harder cases selected using Neural-Net Climate QC 

Easier cases 

Climatology is significantly poorest globally and for all regions 
Version-5 outperforms Version-6 2 Regression MODIS in all spatial regions 
Neural-Net outperforms Version-5 globally and in mid-latitude land and ocean 
Neural-Net is slightly poorer than Version-5 poleward of 50°N 

Harder cases 

Neural-Net is significantly better than all other systems in all regions 
Version-5 is much better than Version-6 2 Regression MODIS in all regions 
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Overall Assessment by SRT 


The Version-6 Neural-Net Start-up option performs significantly better than all others 
in just about every way - including Version-5 
This conclusion was also reached by all speakers at the April 7 Net-Meeting 
The fact that Version-6 Neural-Net boundary layer retrievals are somewhat poorer 
than Version-5 poleward of 50°N is troubling but this is not a show stopper 


Possible contributions to poorer BLT in Version-6 Neural-Net in North Polar region 

• Effect of differences in initial guess 

• Effect of differences in microwave tuning between Version-5 and Version-6 (at JPL) 

- SRT still uses Version-5 microwave tuning 

• Effect of differences in Version-6 retrieval algorithm 

Next figures show Neural-Net boundary layer guess is poorer than Version-5 Clear 
Regression guess, especially poleward of 50°N 
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Comparison of SRT Version-5.44 with JPL 2 Regression MODIS 


SRT Version-5.44 should be scientifically equivalent to JPL 2 Regression MODIS 
except 

• SRT Version-5.44 uses old microwave tuning (like Version-5) 

• SRT Version-5.44 uses old climatology (like Version-5) 

• JPL 2 Regression MODIS is coded differently but meant to be scientifically 
equivalent 

We compare both sets of T(p) retrievals on the easy and hard ensembles 
We compare both sets of QC’d SST’s 

Results show SRT Version-5.44 performs better than JPL 2 Regression MODIS 

Boundary layer temperature is not as bad for harder cases 

Negative SST bias is much less in Version-5.44 than that in JPL 2 Regression 
MODIS and also in JPL Neural Network 
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Recent Changes to Cloud Parameter Retrieval Algorithm 


Experiments conducted were inspired by interaction with Van Dang and 
Evan Manning 

Experiments were conducted using SRT Version-5.44 

Version 5.44 “baseline” performs cloud retrieval exactly as done in JPL Version-5.7.4 

Version 5.44 “new clouds” has 4 changes 

• More damping in the cloud parameter retrieval step 

• Two code changes dealing with treatment of clouds near the surface 

• A code change dealing with first pass cloud retrievals contain only 1 layer 

Results shown are preliminary - this is a work in progress 
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Preliminary Findings 


Compared to Version-5.44 baseline the new cloud retrieval step has 

• Significantly reduced the number of cases with high clouds higher than 120 mb 

This is closer to Version-5 

• Significantly increased the number of cases with low clouds lower than 700 mb 

This is closer to Version-5 

• Decreased cloud fraction (level 1 plus level 2) between 150 mb and 170 mb as 

well as lower than 700 mb - This is closer to Version-5 

• Increased cloud fraction between 170 mb and 550 mb 
These all seem like good things 

New cloud retrieval steps removed all spikes in the cloud distribution as a function of 
pressure 

This is definitely a good thing 
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Required Further Work Before Release of Version-6 


• Code at JPL must be modified to generate error estimates for SCCNN and 
SCCNN AO 

Also needs new tables of coefficients and thresholds (John Blaisdell) 

• New QC thresholds for constituent profiles, total precipitable water, and Clear Sky 
OLR generated using JPL SCCNN and SCCNN AO runs (Lena Iredell) 

• Optimization of QC for C0 2 retrievals using Neural-Net Start-up (Ed Olsen, 

Joel Susskind, ) 

We must have a satisfactory C0 2 product as part of Version-6 

• Modifications to Level 3 code at JPL 

Products in each AIRS FOV should be gridded separately 

Coastal cases (part land, part ocean) should be included in the gridding 

Addition of new parameters to level 3 support product 
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Desired Further Work Before Release of Version-6 


SRT 

Bring up Neural-Net retrieval system (1 month) 

Conduct retrieval optimization studies using Neural-Net system (1-2 months) 
Channel selection and damping parameters for T(p), q(p), skin temperature and 
surface emissivity, cloud clearing and cloud parameters 
Compare results using new and old MW tuning 

CO retrievals - Juying Warner and Eric Maddy 
Install climatology first guess for CO retrieval 
Further study with regard to angle dependence of CO retrievals 

I think new CO RTA needs an empirical correction at large angles 

We might need 3 more months to accomplish the desired research 
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